Supplementary Material for Nonparametric Budgeted Stochastic Gradient Descent

1 Notion We introduce some notions used in this supplementary material. For regression task, we define y max = max y |y|. We further denote the set S as S = B 0, y max λ −1/2 if L2 is used and λ ≤1 R D otherwise where B 0, y max λ −1/2 = w ∈ R D : w ≤ y max λ −1/2 and R D specifies the whole feature space. We introduce five types of loss functions that can be used in our proposed algorithm, namely Hinge, Logistic, L2, L1, and ε−insensitive losses. We verify that these loss functions satisfying the necessary condition, that is, l (w; x, y) ≤ A w 1/2 + B for some appropriate positive numbers A, B. Without loss of generality, we assume that feature domain are bounded, i.e., Φ (x) ≤ 1, ∀x ∈ X. ˆ Hinge loss l (w; x, y) = max 0, 1 − yw T Φ (x) l (w; x, y) = −I {yw T Φ(x)≤1} yΦ (x) Therefore, by choosing A = 0, B = 1 we have l (w; x, y) = Φ (x) ≤ 1 = A w 1/2 + B ˆ L2 loss In this case, at the outset we cannot verify that l (w; x, y) ≤ A w 1/2 + B for all w, x, y. However, to support the proposed theory, we only need to check that l (w t ; x, y) ≤ A w t 1/2 + B for all t ≥ 1. We derive as follows l (w; x, y) = 1 2 y − w T Φ (x) 2 l (w; x, y) = w T Φ (x) − y Φ (x) l (w t ; x, y) = |w T t Φ (x) + y| Φ (x) ≤ |w T t Φ (x) | + y max ≤ Φ (x) w t + y max ≤ A w t 1/2 + B where B = y max and A = y 1/2 max λ −1/4 if λ ≤ 1 y 1/2 max (λ − 1) −1/2 otherwise. Here we note that we make use of the fact that w t ≤ y max (λ − 1) −1 if λ > 1 (cf. Thm. 7) and w t ≤ y max λ −1/2 otherwise (cf. Line 13 in Alg. 2 and Line 16 in Alg. …